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Abstract 

The  need  to  implement  a  truly  agile  acquisition  process  is  apparent.  Current 
acquisition  professionals  are  required  to  brief  decisions  through  the  chain  of  command 
using  a  lengthy  process  to  execute  a  change  in  direction.  Truly  agile  organizations  create 
what  John  Boyd  called  “asymmetric  fast  transients”  in  order  to  maneuver  inside  the 
enemies’  or  competition’s  decision  cycle.  Our  warfighting  doctrine  calls  for  trust  and 
initiative  to  enable  all  levels  of  leadership  to  seize  the  initiative  when  opportunities 
present  themselves.  This  research  presents  the  need  for  development  of  acquisition 
doctrine  that  takes  the  same  approach  in  executing  acquisition  programs.  To  this  end,  an 
innovative  tool  DoD  should  consider  to  reduce  risk  and  shorten  acquisition  cycle  time  is 
the  Perfonnance  and  Reliability  Evaluation  with  Diverse  Information  Combination  and 
Tracking  (PREDICT)  reliability  methodology. 

How  can  PREDICT  help  the  acquisition  process  to  be  more  agile  when  numerous 
acquisition  reform  efforts  of  significant  scope  have  tried  and  failed?  The  unique 
contribution  of  PREDICT  is  using  formal  elicitation  of  expert  knowledge  to  calculate 
concept  reliability  prior  to  testing.  Statistical  analysis  of  the  expert  knowledge  yields  a 
calculation  of  reliability  and  uncertainty  of  the  technology  or  concept.  PREDICT  is 
supporting  the  Los  Alamos  National  Laboratory  (LANL)  mission  of  maintaining  and 
certifying  the  safety  and  reliability  of  nuclear  weapons  without  system  testing.  For  DoD, 
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this  could  provide  a  completely  alternative  methodology  for  detennining  technology 
readiness  levels  and  risk,  before  launching  an  acquisition  effort. 

This  research  will  explain  the  PREDICT  process,  and  then  show  how  its  application 
in  the  DoD  acquisition  process  could  provide  the  following  benefits: 

•  Shorten  the  Technology  Planning  Integrated  Process  Team  (TPIPT)  and 
Modernization  Planning  processes. 

•  Enable  assessment  of  reliability  for  new  concepts  at  Milestone  A  and  for  legacy 
systems  undergoing  change,  eliminating  unproductive  paths  earlier. 

•  Optimize  test  planning  and  execution  by  showing  testing  impact  to  reliability. 

•  Enable  transfer  of  knowledge  as  experienced  personnel  transition  out  of  programs. 

A  classified  Los  Alamos  report  will  document  the  results  of  the  PREDICT  reliability 
calculation  for  the  B61-7,  -11.  Because  of  “need  to  know,”  access  to  the  classified  report 
will  be  detennined  on  a  case-by-case  basis  after  the  initial  reliability  estimate  is 
completed  in  the  fall  of  2005.  A  LANL  point  of  contact  (POC)  is  listed  in  Appendix  D. 


Chapter  1 


Introduction 

The  greatest  challenge  to  any  thinker  is  stating  the  problem  in  a  way  that 
will  allow  a  solution. 

— Bertrand  Russell 

Is  there  a  problem  with  the  DoD  acquisition  process?  The  answer  is  without  a  doubt, 
yes.  The  process  takes  too  long  and  is  too  expensive.  Consider  the  F/A-22  and  the  Space 
Based  Infrared  Satellite  (SBIRS)  weapon  systems  which  were  on  the  drawing  board  when 
the  author  entered  the  Air  Force  in  1986  -  19  years  ago!  The  F/A-22  began  as  the 
Advanced  Tactical  Fighter  in  1984. 1  Likewise,  SBIRS  began  in  1984  as  a  portion  of 
Ballistic  Missile  Defense  called  the  Boost  Surveillance  and  Tracking  System.” 

Both  of  these  programs  have  a  track  record  of  tremendous  delay  and  cost  growth.  In 

1991  the  Air  Force  planned  to  purchase  648  F-22s  for  $86.6  billion.  In  September  1993, 

after  the  Bottom  Up  Review,  the  F-22  purchase  quantity  was  changed  to  442  for  $71.6 

billion.  This  was  a  31  percent  production  quantity  reduction,  but  only  a  17  percent 

reduction  in  cost.  Continuing  this  trend  over  a  number  of  years,  numerous  technical  and 

production  issues,  as  well  as  three  Congressional  budget  cuts  led  to  a  more  recent 

estimate  of  $69.7  billion  for  276  aircraft4  This  equated  to  a  2.6  percent  cost  reduction, 

but  a  38  percent  reduction  in  quantity.  While  the  total  cost  of  the  program  is  capped  by 

Congressional  restriction,  the  per-unit  cost  has  increased  dramatically.  Similarly,  SBIRS- 
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High,  the  early  warning  portion  of  SBIRS,  went  from  an  estimate  of  $  1 .9  billion  in  1996, 
to  an  estimated  $4.5  billion  in  late  2001.'’  Ironically,  these  examples  of  DoD  acquisition 
programs  occurred  during  a  time  of  acquisition  reform,  including  high-profile  efforts 
such  as  the  Lighting  Bolt  initiatives  and  the  elimination  of  military  specifications. 

The  latest  acquisition  reform  push  is  Agile  Acquisition.  The  five  pillars  of  Agile 
Acquisition  are  Collaborative  Requirements,  Technology  Transition,  Seamless 
Verification,  Robust  Systems  Engineering,  and  Expectations  Management.6  A  review  of 
the  five  pillars  reveals  nothing  revolutionary  that  if  fully  implemented  would  have 
reduced  the  cycle  time  for  the  F/A-22  or  SBIRS  from  20  to  5,  or  even  10  years.  For 
example,  Robust  Systems  Engineering  calls  for  designs  that  are  “insensitive  to  variability 
in  manufacturing  and  use.7”  This  implies  exceeding  the  operational  requirements 
specified  by  the  user.  Unfortunately,  the  opposite  is  usually  the  case  -  the  user  is 
normally  asked  to  relax  requirements  in  order  to  reduce  cost  or  meet  production 
schedules.  Also  under  Agile  Acquisition,  the  Air  Force  is  realigning  Program  Executive 
Officer  (PEO)  positions.  Two  PEOs  are  moving  to  Product  Centers  to  be  co-located  with 

o 

their  program  offices.  These  realignments  are  good  and  will  likely  result  in  marginal 
reductions  in  cost  and  cycle  time,  but  nothing  on  the  order  of  a  factor  of  two  or  four,  as 
needed.  What  is  needed  is  a  revolution  in  acquisition  affairs,  not  continued  Band-Aid 
fixes. 

How  does  DoD  move  from  achieving  marginal  reductions  in  cycle  time  to  cutting 
cycle  time  in  half  or  more?  One  thing  not  to  do  is  to  keep  making  adjustments  to  the 
current  system.  In  “Certain  to  Win”  Chet  Richards  explains  the  result  of  partial  fixes, 
“You  won’t  get  there  by  trying  to  do  the  same  stuff,  only  faster.  What  you’ll  get  is  a 
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mess...9”  Despite  the  success  of  our  fielded  forces,  it’s  hard  to  argue  from  a  cost  and 
cycle  time  perspective  that  the  DoD  acquisition  process  is  not  a  “mess.”  A  good 
assessment  of  what  has  occurred  is  described  in  Lt  Col  Steven  Suddarth’s  “Solving  the 
Great  Air  Force  Systems  Irony.”  Suddarth  contends  the  acquisition  bureaucracy  has 
evolved  over  a  period  of  35  years  from  the  “. .  .simple  management  of  complex  systems  to 
the  complex  management  of  simples  ones.”10  Why  has  this  evolution  toward  complex 
management  systems  occurred? 

The  acquisition  disasters  of  the  1970s  and  1980s  are  typified  by  the  example  of  the 
A-12,  or  on  a  smaller  scale,  the  $500  toilet  seat  or  hammer.  The  A-12  Avenger  II  was  a 
Navy  stealth  fighter  that  was  cancelled  in  January  1991  because  of  materials  issues  that 
had  driven  per  unit  cost  to  $165  million.11  Ironically,  the  F/A-22  has  risen  to  $250 
million  per  aircraft,  if  276  units  are  produced  for  $69  billion.  One  reform  effort  after 
another  led  to  tighter  controls  on  decision-making  and  increased  documentation. 
Documentation  does  not  facilitate  communication.  Conversely,  Richards  states, 
“...explicit  forms  are  often  used  not  to  communicate  but  to  document,  not  to  inform  but 
to  protect  oneself.”  ~  Richards  states  mutual  trust  is  critical  to  encouraging  individual 
initiative.  The  institutional  lack  of  trust  in  the  DoD  acquisition  process  has  erased 
individual  initiative.  The  Agile  Acquisition  initiative  is  a  step  in  the  right  direction,  but  it 
appears  it  may  only  be  “trying  to  do  the  same  stuff,  only  faster.”  How  do  we  get  out  of 
the  rut  of  making  marginal  improvements  to  a  process  that  has  delivered  winning 
warfighting  capability,  but  takes  too  long  and  is  too  expensive? 

Trust  and  initiative  are  critical.  Process  and  documentation  have  slowed  the  DoD 
acquisition  process  to  the  point  where  technology  proliferation  through  media  such  as  the 
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Internet  may  enable  our  adversaries  to  apply  and  field  leap-ahead  technologies  before  we 
can.  To  avoid  this  scenario,  chapter  two  discusses  the  origin  of  the  Observe  Orient 
Decide  Act  (OODA)  Loop  decision  process  and  how  it  can  be  applied  in  DoD  acquisition 
to  enable  our  acquisition  professionals  to  seize  the  initiative.  The  idea  of  creating 
Acquisition  Doctrine  and  an  Acquisition  Battlelab  for  special  training  and 
experimentation  are  presented.  Chapter  two  introduces  the  PREDICT  reliability 
methodology  and  discusses  how  a  modified  PREDICT  methodology  could  improve  the 
technology  development  and  modernization  planning  processes.  Chapter  three  briefly 
provides  a  number  of  other  potential  DoD  acquisition  applications  of  PREDICT.  An 
example  of  how  industry  (Delphi  Automotive)  applied  and  validated  the  PREDICT 
methodology  is  provided  in  chapter  four. 

The  appendixes  provide  a  background  on  reliability  theory,  a  more  detailed 
description  of  the  PREDICT  process,  and  a  mock  example  of  applying  the  PREDICT 
reliability  methodology.  Appendix  D  lists  a  LANL  POC  for  requesting  access  to  the 
classified  report  which  will  document  of  the  use  of  PREDICT  on  the  B61-7,  -1 1. 


Notes 

1  Rosario,  Gilberto,  Captain,  USAF,  “The  Advanced  Tactical  Fighter,  The  Joint 
Strike  Fighter,  and  Infonnation  Warfare:  Opportunities  and  Challenges  to  American  Air 
Dominance  in  the  21st  Century.”  Air  &  Space  Power  Chronicles,  22  Mar  99.  On-line. 
Available  at:  http://www.airpower.maxwell.af.mil/airchronicles/cc/rosario.html,  1. 

2  O’Neill,  Malcolm  R.,  Lt.  Gen,  USA  (Director,  Ballistic  Missile  Defense 
Organization),  “Ballistic  Missile  Defense:  12  Years  of  Achievement.”  Statement  to  the 
House  National  Security  Committee,  April  4,  1995.  Defense  Issues:  Volume  10,  Number 
37.  On-line.  Available  at:  http://www.defenselink.mil/speeches/1995/tl9950404- 
oneill.htm,  13. 

3  F-22  Raptor  History,  GlobalSecurity.org,  8  Mar  05.  On-line.  Available  at: 
http  ://www.  globalsecurity .  org/military/ systems/ aircraft/ f-22-cost.htm. 

4  F-22  Raptor  Cost,  GlobalSecurity.org,  8  Mar  05.  On-line.  Available  at: 
http://www.globalsecurity.org/military/systems/aircraft/f-22-cost.htm 


4 


Notes 


5  Richelson,  Jeffrey  T.,  “The  Satellite  Gap.”  Bulletin  of  Atomic  Scientists, 
January/February  2003,  (vol.  59,  no.  01).  On-line.  Available  at: 
http://www.thebulletin.org/article.php?art_ofn=jf03richelson,  4. 

6  Agile  Acquisition,  ACE  CONOPS,  SAF/AQ  &  AFMC/CC,  2  March  04.  On-line. 
Available  at:  http://ax.losangeles.af.mil/axd/agile_acq/agile_acq.htm,  1. 

7  Ibid. 

o 

Agile  Acquisition:  Air  Force  PEO  Realignment  Take  Major  Steps  Forward, 
FindArticles,  On-line.  Available  at: 

http://www.fmdarticles.eom/p/articles/mi_m0QMG/is_2_33/ai_n6005568,  1. 

9  Richards,  Chet,  Certain  to  Win:  The  Strategy  of  John  Boyd,  Applied  to  Business. 
Xlibris  Corporation,  www.Xlibris.com,  2004,  166. 

10  Suddarth,  Steven  C.,  Lt  Col,  USAF,  “Solving  the  Great  Air  Force  Systems  Irony,” 
25  Mar  02.  On-line.  Available  at: 

http://www.airpower.maxwell.af.mil/airchronicles/apj/apj02/spr02/su. . .,  2. 

11  Wikipedia,  A-12  Avenger  II,  1  Apr,  2005.  On-line.  Available  at: 
http  ://en.  wikipedia.  org/wiki/A- 1 2_Avenger_II . 

12  Richards,  120. 


5 


Chapter  2 


OODA  Loop 


Agility:  The  ability  to  change  one ’s  orientation  -  roughly,  worldview  -  in 
response  to  what  is  happening  in  the  external  world. 

— John  Boyd1 

Reducing  DoD  acquisition  cycle  times  by  a  factor  of  two  or  more  will  only  occur 
through  a  revolution  in  acquisition  affairs.  The  acquisition  corps  must  be  equipped  to 
seize  the  initiative  when  opportunities  arise.  Consider  an  Army  company  commander  in 
the  ground  war  during  the  Gulf  War  or  the  War  in  Iraq  (4  days  and  3  weeks  long, 
respectively).  These  officers  were  trained  to  follow  their  “Commander’s  Intent,”  seize 
the  initiative,  move  around  obstacles,  and  push  on  to  accomplish  their  mission.  Imagine 
these  officers  having  to  rewrite  their  operations  orders  and  brief  the  changes  up  the  chain 
of  command  if  they  decided  to  take  hill  B  versus  hill  A  (as  planned)  to  accomplish  their 
mission.  Just  like  a  pilot  switching  from  a  primary  target  to  an  alternate,  these  officers’ 
judgment  is  trusted  and  the  command  and  control  systems  are  in  place  for  the  changes  in 
plans  to  be  seamlessly  executed.  Modern-day  warfighters  make  decisions  on  the  fly, 
keeping  the  enemy  off  balance. 

In  acquisition,  time  is  the  enemy.  Why  not  equip  the  acquisition  professional  with  a 
process  and  the  communication  tools  necessary  to  restore  trust  and  enable  them  to  seize 
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the  initiative?  Applying  the  concepts  of  war  to  the  business  environment  is  not  unheard 
of. 


Asymmetric  Fast  Transients 

In  “Certain  to  Win,”  Richards  applies  Col  John  Boyd’s  OODA  loop  concept  to 
business.  In  the  50’s  Boyd  developed  the  “Aerial  Attack  Study”  which  showed  in  aerial 
combat  for  “every  maneuver  there  is  a  series  of  counter  maneuvers."”  Essentially  the 
same  concepts  are  still  taught  to  fighter  pilots  all  over  the  world.  Col  Boyd  is  credited 

3 

with  designing  the  maneuvering  specifications  for  the  F-16  and  F-18. 

Boyd’s  OODA  loop  concept  grew  out  of  his  study  of  the  F-86  advantage  over  the 
MiG- 15  in  the  Korean  War.  Despite  the  nearly  equal  perfonnance  characteristics,  the  F- 
86  had  a  ten-to-one  victory  ratio  over  the  MiG-15.  Boyd  agreed  pilot  training  was  better 
for  the  F-86,  but  that  “energy  maneuverability”  was  the  driver  for  the  lopsided  record. 
Despite  the  MiG-15’s  tighter  turning  ability,  the  F-86’s  bubble  canopy  provided  better 
visibility,  and  the  F-86’s  full  power  hydraulic  controls  enabled  quicker  aircraft  response. 
These  two  factors  created  what  Boyd  called  “asymmetric  fast  transients.”4  Note  this  did 
not  mean  executing  the  same  maneuvers  quicker.  This  concept  refers  to  transitioning 
from  one  maneuver  to  a  completely  different  or  unexpected  maneuver,  quicker  than  the 
enemy  can  react.  Applied  to  business  or  acquisition,  this  means  avoiding  the  “same  stuff, 
only  faster”  improvement  mentality  and  shifting  to  creating  the  ability  to  quickly  make  an 
abrupt  change  into  an  entirely  new  direction. 

Experience  shows  once  a  concept  and  acquisition  strategy  to  field  a  warfighting 
capability  is  in  place,  it’s  like  shifting  the  orbit  of  a  planet  to  transition  to  a  new 
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approach.  Say,  for  example,  the  program  manager  (PM)  runs  into  a  technology 
roadblock.  The  PM  should  be  able  to  immediately  “maneuver”  around  that  technology 
with  other  approaches  to  field  the  capability.  Likewise,  if  the  PM  recognizes  a 
technology  that  provides  a  shorter  path  to  the  capability,  he  or  she  should  be  able  to 
“seize”  the  initiative  and  pursue  it.  This  level  of  trust  and  personal  initiative  needs  to  be 
reinserted  in  the  acquisition  process.  This  should  be  the  goal  of  Agile  Acquisition. 

How  do  we  evolve  the  acquisition  process  back  to  a  simple  process  managing 
complex  systems?  Project  engineers,  PMs,  and  program  directors  need  to  be  enabled  to 
create  “asymmetric  fast  transients.” 


BLU-118/B 

A  good  example  of  a  team  that  was  enabled  in  this  manner  is  the  team  that  developed 
and  fielded  the  BLU-118/B  Thermobaric  Weapon.  On  11  Oct  01,  the  Defense  Threat 
Reduction  Agency  organized  a  quick-response  team  including  Navy,  Air  Force, 
Department  of  Energy  (DOE),  and  industry.  Their  purpose  was  to  look  at  a  number  of 
on-going  Advanced  Concept  Technology  Demonstrations  and  then  “identify,  test, 
integrate,  and  field  a  rapid  solution  that  would  enhance  weapons  options  in  countering 
hardened  underground  targets.5”  The  Navy  focused  on  development  of  the  new 
explosive,  while  the  Air  Force  had  system  integration,  safety  and  flight,  and  a  modified 
fuze.  On  14  Dec  01,  the  BLU-118/B  was  successfully  tested  at  a  Nevada  Test  Site 
tunnel.  On  21  Dec  01,  Undersecretary  of  Defense  for  Acquisition,  Edward  C.  Aldridge, 
announced  a  small  number  of  the  weapons  were  being  deployed  to  support  the  war  in 
Afghanistan.  In  late  January  02,  the  Air  Force  completed  technical  data  and  flight 
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certifications,  clearing  the  way  for  operational  use  of  the  ten  warheads  that  were 
available.  The  first  combat  use  of  the  warhead  occurred  on  3  Mar  02. 6 

The  team  was  formed  in  mid-October  and  ten  warheads  were  available  for  use  at  the 
end  of  January  -  approximately  three  and  half  months  later.  The  team  created  an 
“asymmetric  fast  transient”  by  modifying  an  existing  penetrator  with  new  capability  to 
reach  the  enemy  where  previously  they  had  been  safe.  The  compressed  process 
employed  on  the  BLU-118/B  initiative  should  be  the  rule,  instead  of  the  exception.  It 
should  not  take  actual  combat  for  the  acquisition  community  to  shift  into  high  gear. 
Whether  we  are  at  war  or  not,  every  delay  drives  two  major  impacts:  1)  the  capability 
delayed  is  not  available  for  use  in  the  field,  and  2)  resources  for  addressing  other 
warfighting  capabilities  are  not  available. 

Acquisition  Doctrine 

Our  vision  should  be  to  field  the  warfighters’  needs  for  the  Global  War  on  Terrorism 
(GWOT)  in  the  shortest  cycle  time  possible.  Time  wasted  on  twenty-year  long 
acquisition  efforts  could  very  well  equate  to  lives  lost.  The  time  has  come  for  the 
development  of  Acquisition  Doctrine  to  guide  the  equipping  and  training  of  the 
acquisition  corps  for  Agile  Acquisition.  A  first  step  should  be  the  formation  of  an 
Acquisition  Battlelab.  Here  an  initial  cadre  of  special  acquisition  forces  could  attend  the 
acquisition  equivalent  of  the  “weapons”  school.  The  focus  should  be  on  fielding 
technologies  to  meet  the  Combatant  Commander’s  highest  priority  shortfalls  in 
prosecuting  the  GWOT.  Acquisition  cycle  times  of  one  month,  one  year,  or  five  years  for 
more  complex  efforts  should  be  the  norm.  In  parallel,  this  Acquisition  Battlelab  would 
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be  responsible  for  developing  the  Acquisition  Doctrine  of  fast  cycle  times  and  planning 
the  transformation  of  the  entire  acquisition  corps  in  ten  years. 

OODA  Loops  and  Planning 

At  the  Air  Force  major  command  (MAJCOM)  level,  experts  from  the  Air  Force 
Research  Lab  (AFRL),  the  Product  Centers,  Combatant  Commands,  System  Program 
Offices,  and  industry  support  a  two-year  modernization  planning  process.  The  output  of 
this  process,  at  least  for  Air  Force  Space  Command  (AFSPC),  is  a  Strategic  Plan  which 
supports  the  development  of  their  Program  Objective  Memorandum  (the  six-year  plan  for 
funding  operations  and  acquisition).  Note  that  AFSPC  experimented  with  a  one-year 
process,  but  is  likely  returning  to  a  two-year  process.  The  MAJCOMs  use  different  terms 
for  the  process  including  strategic  planning,  the  modernization  planning  process,  and  the 
integrated  planning  process.  The  latest  term  for  Air  Force  level  strategic  planning  is  the 
Capabilities  Based  Planning  process.  Although  there  are  different  names,  the  goal  is  to 
identify  capabilities  to  fill  the  gaps  in  warfighting  ability. 

AFRL  and  the  Product  Centers  support  MAJCOM  modernization  planning  with  their 
TPIPTs.  The  TPIPTs,  in  cooperation  with  industry,  attempt  to  answer  the  question  of 
how  technology  can  be  used  to  fill  gaps  in  military  capability.  According  to  Dr.  (Lt  Col) 
Russ  McNutt,  the  TPIPTs  process  is  broken  -  too  many  people  involved  and  it  takes  too 
long.7  The  process  needs  large  numbers  of  people  to  meet  numerous  times  over  the 
course  of  two  years  in  their  effort  to  write  technology  development  plans. 

However,  participation  is  limited,  at  least  for  the  modernization  planning  process 
(based  on  the  author’s  personal  experience).  Essentially  what  occurs  is  a  small  group 

ends  up  doing  the  very  best  they  can  with  limited  input  from  the  numerous  stakeholders. 
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No  “stick  in  the  eye”  is  intended  -  tightened  travel  budgets  and  personnel  reductions  have 
a  major  impact  on  these  types  of  long-term  travel  intensive  planning  efforts.  Two  years 
is  an  unacceptable  cycle  for  developing  technologies  and  concepts  to  fight  the  GWOT. 
Most  importantly,  a  two-year  planning  cycle  very  likely  puts  the  terrorist  inside  our 
planning  OODA  loop.  How  could  this  two-year  cycle  be  reduced  by  a  factor  of  two,  or 
perhaps,  four? 

Expert  Input 

Tremendous  expertise  resides  in  DoD  and  industry.  As  mentioned  above,  a  key 
problem  in  the  planning  process  is  participation  by  the  experts.  How  do  you  harness  this 
expertise  and  bring  it  together?  The  appendixes  of  this  research  describe  in  detail  and 
provide  an  example  of  a  methodology  LANL  is  using  to  formally  elicit  and  quantify 
expert  judgment  to  calculate  concept  reliability  for  nuclear  weapons  prior  to  testing.  The 
methodology  is  called  PREDICT  -  Perfonnance  and  Reliability  Evaluation  with  Diverse 
Information  Combination  and  Tracking.  PREDICT  is  defined  as  a  “set  of  formal, 
structured  techniques  for  eliciting,  quantifying,  and  analyzing  an  often-undocumented 

g 

asset:  the  knowledge  that  resides  with  a  company’s  designers,  engineers,  and  scientists.” 
For  LANL,  the  issue  was  calculating  reliability  early,  during  the  concept  phase,  and 
quantifying  reliability  without  a  system  test.  For  DoD,  the  PREDICT  methodology  could 
be  adapted  to  implement  virtual  TPIPTs  (VTPIPTs)  and  virtual  modernization  planning 
efforts,  eliciting  expert  knowledge  from  participants  spread  out  across  the  country. 
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Modified  PREDICT  Methodology 


Informed  decision-making  when  little  data  /  information  is  available  is  key.  At 
LANL,  the  team  was  on-site  and  participated  on  a  non-interference  basis  (with  their 
primary  duties)  over  a  long  period  of  time.  The  author  participated  for  eight  months  on 
the  PREDICT  team  that  met  twice  a  week  for  one  and  a  half  hour  elicitation  sessions  with 
the  scientists  and  engineers  that  designed  the  changed  components  for  the  LEP  of  the 
B61-7,  -11.  For  LANL,  reliability  of  changes  to  existing  weapons  is  key.  These 
weapons  cannot  be  tested  at  the  system  level,  so  a  detailed  and  somewhat  lengthy  process 
is  employed  to  ensure  reliability.  A  thorough  discussion  of  reliability  theory  and  the 
PREDICT  process  are  provided  in  Appendix  A  and  B,  respectively.  The  essence  of 
PREDICT  is  a  common  sense  step-by-step  methodology  for  rigorously  extracting 
(eliciting)  knowledge  from  subject  matter  experts  (SMEs)  and  then  quantifying  the  data 
so  decision-makers  can  make  informed  decisions. 

The  key  application  for  DoD  is  recognizing  the  value  of  analyzing  expert  knowledge 
to  aid  decision-making.  A  critical  issue  in  technology  development  for  the  TPIPTs  is 
understanding  the  maturity  of  a  given  technology.  A  modified  PREDICT  methodology 
could  be  used  in  the  TPIPTs  process  to  better  characterize  technology  maturity  and  draw 
in  greater  participation. 


VTPIPT 

Establishing  a  VTPIPT  would  not  be  “doing  the  same  stuff,  only  faster.”  This  would 
be  a  completely  new  direction.  Consider  a  team  of  recognized  military,  government 
civilian,  and  academic  experts  in  a  specific  mission  area  /  technology.  Using  networking 
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technologies,  the  core  TPIPT  would  elicit  information  by  electronically  sending  out  their 
requests  for  information  on  technology  concepts  that  could  fill  gaps  in  military  capability. 
Instead  of  traveling  to  meetings,  these  experts  would  reply  electronically.  Memorandas 
of  Agreement  would  establish  expectations  for  response  with  government  experts  - 
message  boards  and  websites  would  facilitate  flow  of  information.  Academic  experts 
would  be  on  contract  to  reply  within  a  specified  time.  The  core  team  would  minimize 
bias  by  using  the  Delphi  method.  With  the  Delphi  method,  the  experts  submit  ideas 
independently.  In  a  group  setting,  personality  and  position  can  easily  bias  a  decision. 
With  a  VTPIPT,  replies  to  questionnaires  (elicitations)  would  be  independent  between  the 
submitters.  As  the  core  team  synthesizes  the  data,  subsequent  rounds  of  questionnaires 
would  update  everyone  on  the  good  ideas  received,  so  the  entire  team  could  see  and 
comment  on  the  progress  of  the  analysis. 


Trust 

The  entire  focus  of  any  DoD  use  of  the  PREDICT  process  should  be  to  create 
“asymmetric  fast  transients”  to  speed  up  the  acquisition  OODA  loop.  To  do  so,  lower 
level  acquisition  leaders  need  to  be  able  to  make  decisions  and  move  out,  without  having 
to  go  through  the  enormous  briefing  and  documentation  burden  currently  required.  In 
war,  leadership  at  the  strategic,  operational,  and  tactical  level  is  trusted  to  make  life  or 
death  decisions  that  affect  the  outcome  of  the  mission.  In  the  current  DoD  acquisition 
process,  leadership  is  not  trusted,  the  process  is. 

PREDICT  is  one  of  potentially  many  decision  tools  the  Acquisition  Battlelab  should 
experiment  with  to  reduce  cycle  time.  Success  should  be  evaluated  based  on  significantly 


reducing  the  time  from  when  the  warfighter  identifies  the  need  to  when  the  solution  is  in 
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hand  to  fill  the  gap  in  military  capability.  Other  tools  should  eventually  be  employed  for 
all  levels  of  the  acquisition  process.  Again,  the  complex  processes  in  place  evolved  over 
35  years.  Twenty  years  of  acquisition  reform  has  resulted  in  twenty  year  cycle  times  for 
major  systems.  The  Acquisition  Battlelab  should  be  chartered  to  continuously 
experiment  with  technology  and  processes  to  reduce  cycle  time  and  to  develop 
acquisition  doctrine  to  guide  the  training  and  equipping  of  our  acquisition  professionals. 

Trust  and  initiative  need  to  be  reestablished  at  all  levels  so  opportunities  can  be 
seized  at  the  earliest  point.  If  a  PM  wants  to  make  a  change  in  direction,  ideally  a 
communications  capability  should  be  in  place  to  make  the  notification  and  get  the  green 
or  red  light  from  the  chain  of  command  within  minutes  or  hours.  Notification  levels 
could  be  established  which  would  correspond  to  the  tactical,  operational,  or  strategic 
level  of  acquisition.  Establishing  guidelines  for  these  levels  of  acquisition  would 
facilitate  deciding  how  high  up  the  chain  of  command  a  given  decision  needs  to  go. 
While  the  BLU-1 18/B  example  was  a  rather  straightforward  non-complex  effort,  the  team 
must  have  established  streamlined  communications  methods  to  keep  the  effort  on  track. 
Whether  on  land,  sea,  or  in  the  air,  our  fighting  doctrine  is  centered  on  seizing  the 
initiative  and  maneuver  at  all  levels  of  war.  Similar  doctrine  should  be  developed  for 
DoD  acquisition  to  support  decision-making  for  significantly  reduced  cycle  times. 

Chapter  three  looks  at  other  potential  applications  of  PREDICT  in  DoD  acquisition 
including  how  PREDICT "s  emphasis  on  expert  knowledge  fits  well  with  Knowledge- 
Based  Acquisition. 
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Chapter  3 


Other  DoD  Acquisition  Applications  of  PREDICT 


Knowledge-based  acquisition  is  a  management  approach  which  requires 
adequate  knowledge  at  critical  junctures  (i.e.,  knowledge  points) 
throughout  the  acquisition  process  to  make  informed  decisions. 

— DoD  Directive  5000.1 

The  acquisition  OODA  Loop  must  be  sped  up  significantly.  Only  changes  that 
reduce  cycle  time  should  be  incorporated.  This  section  briefly  looks  at  a  number  of  other 
areas  in  the  DoD  acquisition  process  which  PREDICT  could  be  used  to  reduce  cycle 
time.  These  areas  include: 

•  Knowledge-based  Acquisition. 

•  Concept  Refinement. 

•  Test  program  development  and  execution. 

•  Documenting  program  history. 

•  Tracking  system  change. 

Knowledge-based  Acquisition 

DoD  Directive  5000.1  “calls  for  sufficient  knowledge  to  reduce  the  risk  associated 
with  program  initiation,  system  demonstration,  and  full-rate  production.”1  The  directive 
goes  on  to  say  the  following: 

Implicit  in  this  approach  is  the  need  to  conduct  the  activities  that  capture 
relevant,  product  development  knowledge.  And  that  might  mean 
additional  time  and  dollars.  However,  knowledge  provides  the  decision 
maker  with  higher  degrees  of  certainty,  and  enables  the  program  manager 
to  deliver  timely,  affordable,  quality  products." 
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To  make  a  significant  improvement,  simply  delete  the  sentence  “And  that  might  mean 
additional  time  and  dollars.”  Then  add  a  statement  at  the  end  of  the  quote  that  reads 
“Any  tools  or  processes  incorporated  should  enable  lower  level  decision-making  to 
restore  trust  and  initiative  in  the  DoD  Acquisition  Process.”  PREDICT  enables  the 
systematic  capture  of  subject  matter  expertise  /  knowledge,  early  and  throughout  the 
acquisition  process.  This  is  the  key  to  timely  lower  risk  decision-making  by  the  PM  and 
the  Milestone  Decision  Authority  (MDA). 


PREDICT  at  Concept  Refinement 


PREDICT  could  be  implemented  during  Concept  Refinement  -  for  reference,  the 
Defense  Acquisition  Management  Framework  is  provided  in  figure  one  . 
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Figure  1  Defense  Acquisition  Management  Framework 

The  purpose  of  the  Concept  Refinement  Phase  is  to  refine  the  initial  concept  and 
develop  a  Technology  Development  Strategy.  Concept  refinement  is  accomplished  by 
conducting  an  Analysis  of  Alternatives  (AoA)  for  the  selected  concept.  This  concept  is 
identified  in  the  Initial  Capabilities  Document  which  documents  gaps  in  military 
capability.  The  AoA  assesses  critical  technology  maturity  and  risk  associated  with  the 

selected  concept  to  determine  the  best  alternative  to  close  the  gap  in  military  capability.4 
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The  same  systems  engineers  and  designers  that  conduct  the  AoA  could  also  feed  the 
PREDICT  process.  The  PREDICT  team  would  conduct  the  elicitations  and  translate  the 
expert  knowledge  into  a  reliability  estimate.  The  focus  at  this  level  would  be  on  the 
experts’  knowledge  and  experience  with  the  critical  enabling  technologies  for  the 
concept.  PREDICT  could  also  determine  the  impact  to  reliability  from  adding  or  deleting 
requirements  to  the  system,  which  occurs  frequently  early  in  the  acquisition  process. 
This  data  would  be  included  in  the  AoA  results. 

A  search  of  the  DoD  Acquisition  Guidebook  (DoDI  5000.2)  reveals  the  first  time  an 
actual  estimate  of  reliability  is  currently  required  is  in  support  of  the  Design  Readiness 
Review,  during  the  System  Development  and  Demonstration  Phase.5  PREDICT  would 
move  this  forward  significantly  by  supporting  an  estimate  of  reliability  as  exit  criteria  for 
the  Concept  Refinement  Phase  at  the  milestone  A  decision.  PREDICT  could  uncover 
“asymmetric  fast  transients”  the  AoA  team  and  /  or  the  MDA  could  leverage  to  reduce 
the  overall  risk  of  the  concept  being  successfully  implemented  for  the  warfighter. 

Reduced  Test  Program  Risk 

PREDICT  can  also  reduce  risk  in  the  development  and  execution  of  test  programs. 
How?  By  highlighting  challenging  technologies  and  /  or  worst-case  performance 
environments,  noting  where  reliability  is  low  and  uncertainty  is  high.  As  shown  in  the 
example  in  Appendix  C,  PREDICT  enables  the  test  developer  to  focus  limited  resources 
on  areas  of  concern  or  uncertainty.  This  enables  better  decision-making,  just  like  a 
reconnaissance  team  locating  enemy  forces  on  the  battlefield,  so  the  commander  has  the 
knowledge  to  maneuver  and  seize  the  initiative.  Further,  with  PREDICT,  the  PM  can 


defend  the  test  program  by  showing  the  impact  to  reliability  and  uncertainty  if  specific 
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tests  are  eliminated  because  of  funding  cuts,  for  example.  Likewise,  the  PM  can  defend 
the  need  for  additional  testing  by  showing  the  impact  to  reliability  if  a  specific  test  is 
added  to  the  program. 


Program  History 

Documentation  is  critical  to  successful  implementation  of  PREDICT.  The 
systematic  capture  and  documentation  of  expert  knowledge  throughout  the  system 
lifecycle  would  be  invaluable  as  personnel  transition  in  and  out  of  the  program.  This 
could  be  likened  to  the  acquisition  community’s  intelligence  preparation  of  the 
battlefield.  One  can’t  make  timely  decisions  without  a  solid  knowledge  of  the  program 
issues,  past  and  present.  The  PREDICT  database  would  be  a  ready  source  of  information 
and  history  for  current  and  new  personnel  to  review  the  technical  basis  for  previous 
decisions. 

Why  would  PREDICT  succeed  where  others  have  failed  at  providing  an  effective 
transfer  of  program  history?  The  bottom-line  is  the  “history”  PREDICT  provides  is  not 
just  a  history,  it’s  a  documented  track  record  of  system  performance!  In  a  typical 
program  office,  when  the  tasker  arrives  to  provide  the  input  to  the  history  office,  the  boss 
looks  around  and  tags  the  junior  officer  or  civilian.  The  likely  input  is  simply  a  timeline 
of  events,  without  the  technical  information  and  data  to  backup  the  significant  events  and 
decisions.  As  discussed  in  Appendix  B,  PREDICT  documentation  must  provide 
traceability  giving  the  Who,  What,  When,  and  Why  a  given  decision  was  made. 

The  documentation  from  a  properly  implemented  PREDICT  methodology  will  be  in 
high  demand.  The  MDA,  PM,  test  developers,  AoA  study  team  members,  and  yes,  the 

action  officer  building  the  history  office  input  will  all  leverage  this  documentation. 
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System  Change 


As  changes  are  made  to  fielded  systems  to  meet  the  evolving  needs  of  the  warfighter 
or  to  address  aging  issues,  PREDICT  can  be  used  to  evaluate  the  changes  based  on 
system  reliability  impacts,  again,  reducing  risk  in  decision-making.  This  applies  to 
systems  in  which  PREDICT  was  implemented  at  Concept  Refinement,  or  when  changes 
to  a  legacy  system  were  being  considered.  Whether  used  on  a  new  or  legacy  system,  with 
PREDICT,  reliability  is  continuously  tracked  as  the  system  evolves  providing  a 
documented  technical  and  reliability  history  of  key  program  changes. 

A  key  question  to  ask  is  whether  or  not  PREDICT  itself  is  reliable?  Why  should 
DoD  consider  a  methodology  without  some  assurance  of  success?  Chapter  four  answers 
this  question  with  a  look  at  Delphi  Automotive’s  application  of  PREDICT. 


Notes 

',  The  Defense  Acquisition  System,  DoD  Directive  5000.1,  USD  (AT&L),  12  May, 
2003,  paragraph  1 1.5.  On-line.  Available  at: 

http://akss.dau.mil/dag/DoD5000.asp?view=document&rf=GuideBook\IG_c  1 1 .5. asp 

2  Ibid,  paragraph  1 1.5. 

3  DoD  Acquisition  Guidebook,  DoD  Instruction  5000.2,  USD  (AT&L),  12  May, 
2003,  paragraph  3.  On-line.  Available  at: 

http://akss.dau.mil/dag/DoD5000.asp?view=document&rf=DoD5002\Figurel.asp. 

4  Ibid,  paragraph  3.5. 

5  Ibid,  paragraph  3.7.4. 
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Chapter  4 


Commercial  Application  of  PREDICT 


"PREDICT  allows  our  customers  to  leverage  our  company ’s  greatest  asset 
-  the  expert  knowledge  of  our  people. " 

— William  S.  Warren,  Director  of  Engineering  for  Delphi  Energy  &  Engine 

Management  Systems1 

The  commercial  application  of  PREDICT  is  limited.  The  author  was  only  able  to 
find  one  commercial  use  of  PREDICT,  a  Delphi  Automotive  fuel  system  application. 
Delphi  worked  in  collaboration  with  LANL  to  develop  PREDICT  beginning  in  the  mid- 
1990s  for  a  period  of  approximately  five  years.  Note  that  in  1999  LANL  and  Delphi 

were  jointly  recognized  with  an  “R&D  100  Award”  from  R&D  Magazine,  a  yearly 

2 

recognition  of  the  100  most  technologically  significant  products  and  processes. 

Delphi 

Delphi’s  interest  was  driven  by  the  major  expense  of  product  changes.  One  problem 
corrected  during  manufacturing  cost  $300,000.  If  the  problem  was  caught  later  and 
resulted  in  a  recall,  it  could  cost  $5, 000, 000. 3  As  discussed  earlier,  LANL’s  interest  was 
in  developing  a  methodology  to  combine  expert  knowledge  with  sparse  test  data  to  better 
calculate  reliability  of  weapons  undergoing  change.  This  common  interest  led  to  Delphi 
working  with  LANL  to  develop  the  PREDICT  reliability  methodology. 
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In  the  commercial  application,  Delphi  established  teams  throughout  the  world  that 
focused  on  using  PREDICT  to  improve  a  fuel  injection  system.  Experts  answered 
formalized  questions  in  ranges  that  could  be  quantified  and  graphed  using  probability 
distributions,  as  discussed  in  the  gas  valve  example  (Appendix  C).  These  distributions 
were  combined  using  Monte  Carlo  simulations.  The  results  focused  testing  on  risky  areas 
to  improve  reliability  and  reduce  uncertainty.  Without  testing,  engineers  could  answer 
“what  if’  questions  about  specific  portions  of  the  design  to  determine  if  a  potential  design 
change  had  a  significant  impact  on  reliability.4  The  goal  was  to  ask  questions  and  make 
changes  until  the  system  met  or  exceeded  the  reliability  requirement. 

Reliability  of  PREDICT 

How  do  we  know  the  methodology  works?  For  DOE  weapons,  system  level  tests 
cannot  currently  be  performed  to  corroborate  PREDICT  estimates.  Even  if  testing  was 
allowed,  the  number  of  tests  required  to  be  statistically  significant  would  very  likely  not 
be  approved.  However,  at  Delphi,  tests  were  conducted  and  customer  data  was  collected 
subsequent  to  the  estimates  made  with  PREDICT.  Delphi  set  specific  reliability  goals  for 
PREDICT  related  to  fuel  injection  warranty  work  and  product  recalls.  The  data  collected 
corroborated  the  PREDICT  calculations  prior  to  testing.  This  was  not  just  one  isolated 
example  at  Delphi.  PREDICT  was  used  on  five  concept  Delphi  systems.5 

Unfortunately  the  use  of  PREDICT  was  not  continued  after  a  change  of  ownership  at 
Delphi  led  to  new  leadership  and  management.  As  is  always  the  case  with  new  and 
innovative  processes,  a  solid  commitment  from  leadership  is  required  to  maintain 
momentum. 
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Notes 


1  Delphi  and  Los  Alamos  National  Laboratory  Receive  Prestigious  R&D  100  Award. 
The  Auto  Channel,  24  September,  1999.  On-line.  Available  at: 
http://www.theautochannel.com/news/press/date/19990924/press000042.html,  1. 

2  Ibid,  1. 

3  R&D  100  Award,  Los  Alamos  National  Laboratory.  LALP-99-184,  August,  1999. 
On-line.  Available  at: 

http://www.stat.lanl.  gov/projects/projects/ Army/Section_l/R&D.pdf. 

4  Performance  and  Reliability  Evaluation  with  Diverse  Information  Combination  and 
Tracking,  Program  Treatment  (text  for  Delphi  DVD  promo),  18  February,  1999,  5. 

5  Booker,  Jane  M.,  Thomas  R.  Bement,  Mary  A.  Meyer,  William  J.  Kerscher. 
PREDICT:  A  New  Approach  to  Product  Development  and  Lifetime  Assessment  Using 
Information  Integration  Technology.  Los  Alamos  National  Laboratory  document:  LAUR- 
00-4737,  May  2003,  5. 
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Chapter  5 


Conclusions 

An  expert  knows  all  the  answers  -  if  you  ask  the  right  questions. 

— Author  Unknown 

After  twenty  years  of  reform  initiatives,  it’s  time  to  move  beyond  “the  same  stuff, 
only  faster”  approach  to  acquisition  reform.  Acquisition  doctrine  should  be  developed 
that  feeds  off  our  warfighting  doctrine  which  calls  for  trust  and  initiative  at  all  levels. 
The  acquisition  OODA  Loop  must  be  shortened  by  implementing  decision  tools  like 
PREDICT  to  create  “asymmetric  fast  transients.”  Decision  tools  emphasizing  trust  and 
initiative  over  documentation  and  process  should  be  put  in  place.  A  DoD  Acquisition 
Battlelab  is  one  option  to  consider  as  a  change  agent  for  at  least  a  ten-year  effort  to 
transform  DoD  acquisition  to  be  truly  agile,  focused  on  the  vision  of  filling  Combatant 
Commander  gaps  in  military  capability  in  the  shortest  cycle  time  possible.  The 
Acquisition  Battlelab  could  also  be  the  initial  location  for  the  equivalent  of  a  “weapons” 
school  for  the  acquisition  special  operators  that  are  employed  there. 

The  key  to  any  acquisition  reform  initiative  should  be  creating  John  Boyd’s 
“asymmetric  fast  transients.”  Acquisition  professionals  need  the  ability  to  divert  off 
unproductive  paths  immediately.  Communications  should  move  up  and  down  the  chain 
of  command  quickly,  similar  to  the  modern  battlefield,  with  levels  of  acquisition 
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mirroring  the  tactical,  operational,  and  strategic  levels  of  war.  The  BLU-118/B  case 
demonstrates  we  have  the  ability  to  be  agile  if  motivated.  Every  delay  in  today’s 
constipated  acquisition  process  has  a  huge  expense  when  one  considers  the  marching 
anny  behind  an  acquisition  effort.  The  DoD  acquisition  process  is  fighting  the  equivalent 
of  WWI  trench  warfare  when  it  should  be  engaged  in  the  agile  maneuvers  of  the  Afghan 
and  Iraq  wars. 

The  PREDICT  methodology  could  be  one  of  the  first  experiments  the  Acquisition 
Battlelab  applies.  A  number  of  potential  applications  for  PREDICT  in  the  DoD 
acquisition  process  were  discussed: 

•  Shorten  the  TPIPTs  and  Modernization  Planning  processes. 

•  Enable  assessment  of  reliability  for  new  concepts  at  Milestone  A  and  for  legacy 
systems  undergoing  change,  eliminating  unproductive  paths  earlier. 

•  Optimize  test  planning  and  execution  by  showing  testing  impact  to  reliability. 

•  Enable  transfer  of  knowledge  as  experienced  personnel  transition  out  of  programs. 

LANL  and  Delphi  implemented  PREDICT  at  the  sub-system  level.  This  may  be  the 
best  approach  for  DoD  to  experiment  with  the  implementation  of  PREDICT  or  other 
tools  to  create  “asymmetric  fast  transients.”  A  system-level  application  would  be  a  huge 
effort  for  any  new  decision  tool.  Legacy  systems  are  continually  undergoing  change  to 
address  aging  issues  and  to  keep  up  with  the  evolving  threat.  A  modernized  guidance  set, 
communications  suite,  or  sensor,  for  example,  would  be  prime  candidates  for  the  first  use 
of  PREDICT  in  DoD  acquisition.  This  approach  would  start  small  and  expand  as  the 
benefits  of  PREDICT  or  any  new  decision  tools  are  proven. 

It’s  time  to  lessen  the  burden  on  the  acquisition  corps.  When  considering  cockpit 
design,  a  key  issue  is  task  saturation.  The  PM  is  beyond  task  saturation  -  that’s  why  it 
takes  twenty  years  to  field  next  generation  systems  like  the  F/A-22  or  SBIRS.  The 
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motivation  for  change  should  be  freeing  up  resources  for  other  acquisition  efforts 
supporting  the  GWOT.  Shortened  cycle  time  equates  to  fewer  gaps  in  military  capability. 
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Appendix  A 


Reliability  Background 


These  appendixes  are  intended  to  provide  a  thorough  discussion  of  how  LANL  is 
using  the  PREDICT  reliability  methodology.  The  key  point  for  the  application  of 
PREDICT  to  reduce  DoD  acquisition  cycle  time  is  the  idea  of  eliciting  expert  knowledge 
as  a  decision  tool,  when  information  is  sparse. 

Before  looking  at  the  PREDICT  process  and  the  example,  Appendix  A  provides  a 
technical  foundation  by  describing  mission  reliability,  uncertainty,  key  reliability 
equations,  and  Bayes’  Theorem. 


Mission  Reliability 

For  “single-shot”  systems  like  a  bomb,  mission  reliability  is  specified1  to  define 
performance.  Mission  reliability  is  defined  as  “the  probability  that  a  system  will  perform 
mission  essential  functions  for  a  period  of  time  under  the  conditions  stated  in  the  mission 
profile.”  The  stated  reliability  would  apply  for  a  specified  period  of  time;  for  example, 
the  life  of  the  bomb,  which  could  be  5,  10,  or  20  years.  The  cost  of  poor  mission 
reliability  is  additional  sorties,  increased  operational  costs,  and  greater  potential  for 
casualties. 
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Uncertainty 


To  determine  the  level  of  confidence  in  the  reliability  estimate,  the  amount  of 
uncertainty  must  be  detennined.  Uncertainty  arises  from  two  sources.  The  first  is  the 
scatter  or  variability  of  data  resulting  from  actual  tests.  The  second  source  of  uncertainty 
reflects  the  expert’s  lack  of  knowledge,  the  things  they  don’t  know,  or  differences  of 
opinion.  Each  expert,  for  example,  will  provide  different  responses  based  on  their  own 

3 

set  of  assumptions  gained  from  their  life  experiences. 

If  a  number  of  experts  are  asked  the  same  question,  there  will  be  differences  in  their 
replies,  just  as  there  is  variability  in  the  results  from  running  the  same  test  multiple  times. 
Using  PREDICT,  these  differences  in  expert  knowledge  are  analyzed  just  as  if  you  were 
assessing  test  data  results.  Some  experts  will  quantify  their  estimates  with  specific 
numbers  or  ranges.  Others  may  use  terms  such  as  high  or  low,  which  are  quantified  with 
numbers  such  as  a  .9  or  .1,  for  example,  for  use  in  calculations.  Once  the  differences  in 
expert  knowledge  are  quantified,  the  mean  and  standard  deviation  are  calculated.  The 
reliability  estimate  is  the  mean,  while  the  corresponding  uncertainty  is  the  standard 
deviation.  A  rule  of  thumb  in  reliability  theory  is  to  use  two  times  the  standard  deviation 
or  the  95  percent  confidence  interval  for  a  conservative  uncertainty  estimate.4 

How  do  you  transition  from  uncertainty  to  confidence?  Confidence  can  be  thought 
of  as  inversely  related  to  uncertainty,  as  shown  with  the  blue  line  in  figure  two.  If  you 
have  high  uncertainty,  there  is  low  confidence  in  the  reliability  estimate  (red  line,  figure 
two).  Conversely,  if  you  have  low  uncertainty,  there  is  high  confidence  in  the  reliability 
estimate  (green  line,  figure  two).5  These  terms  sound  straightforward,  but  the 
calculations  to  derive  them  can  quickly  become  involved. 
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Figure  2  Confidence  vs.  Uncertainty 
Calculations 

The  calculations  for  determining  the  reliability  of  a  system  are  driven  by  how  the 
subsystems  are  interconnected  and  their  relationships  to  one  another.  A  system  structure 
can  be  depicted  with  a  series  model,  a  redundant  (parallel)  model  or  a  mixed  model.  The 
following  notes  apply  to  the  figures  below  which  depict  each  of  these  models. 

•  Reliability  formula  and  assumptions  are  shown  for  each  model 

•  Rs  =  Reliability  of  the  system 

•  Ri  =  Reliability  of  the  ith  subsystem 

•  Qs  =  1  -  Rs  -  unreliability  of  the  system 

•  Qi  =  1  -  Ri  =  unreliability  of  the  ith  subsystem 

•  n  =  Product  of 


Reliability^  =  R1Rj...RIi=  nFl 
i=l 

Assumes  independent  subsystems 


Figure  3  Series  Model 


29 


n  n 

Unreliability^  (Qj)  =  QjQ2  Qa=  nQj  =  nfl-R^ 

i=l  i=l 
n 

Reliabilitys  =  1  -  Q,  =  1  -  n  (1-R^ 
i=l 

Assumes  only  one  ofn  subsystems  be  operating  for 
mission  success 


Figure  4  Redundant  Model 


Reliability^  I^RjRjtl  -  (1  -  R4)(l  -  Rj)(l  -  R()] 

Assumes  independent  subsystems  and  only  one  redundant 
subsystem  must  be  operating  for  mission  success 


Figure  5  Mixed  Model 

These  models  reflect  the  simplest  form  of  the  system  structures  discussed.  As  noted  in 

the  figures,  the  models  assume  independent  subsystems.6 

How  are  uncertainty  and  confidence  determined  for  the  system?  The  reliability  of 

each  element  in  the  model  has  variability,  whether  the  data  source  was  actual  test  results 

or  expert  knowledge.  For  each  element,  the  mean  and  standard  deviation  are  calculated 
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and  then  combined  mathematically,  using  a  Monte  Carlo  simulation,  for  example,  to 
calculate  the  mean  and  standard  deviation  for  the  system.7 

Monte  Carlo  simulation  was  originally  developed  for  the  Manhattan  Project  in  World 
War  II.  The  simulation  uses  random  numbers  and  statistics  to  solve  complex 
mathematical  problems.  For  each  variable  being  modeled,  a  probability  distribution  is 
assigned  for  the  range  of  possible  values.  The  simulation  randomly  generates  values  for 

the  variables  according  to  the  distributions  specified.  The  simulation  is  run  over  and  over 

8 

(i.e.,  10,000  times)  to  generate  an  answer  to  the  problem. 

Bayes’  Theorem 

How  does  PREDICT  combine  different  sources  of  infonnation,  such  as  historical 

data,  test  data,  and  expert  knowledge?  A  key  infonnation  combination  tool  used  in 

PREDICT  is  Bayes’  Theorem,  shown  below. 

Prob  (AYB)  =  Prob  (A)  Prob  (B\A)  /  Prob  (B) 

The  Prob  (AYB)  is  read  as  the  probability  of  A  given  that  B  occurred.  Likewise,  Prob 

(B\A)  is  read  as  the  probability  of  B,  given  A  occurred.  These  two  probabilities  are 

conditional,  meaning  they  measure  the  degree  of  plausibility  of  the  event,  given  the  other 

occurred.  The  probability  of  A  and  B  (Prob  (A)  and  Prob  (B))  are  unconditional 

probabilities  in  that  they  are  not  tied  to  the  other  event  occuning  or  not  occurring.9 

Bayes’  Theorem  is  used  because  it  recognizes  the  value  of  mathematically 

combining  prior  information  with  new  information  (likelihood)  to  reduce  uncertainty.10 

In  most  cases,  there  is  some  prior  knowledge  (i.e.,  historical  data)  that  can  be  quantified 

as  a  “prior  distribution.”  A  “likelihood  distribution”  is  calculated  from  the  elicitations, 

and  combined  with  the  prior  distribution  to  form  the  “posterior  distribution.”11 
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In  a 


successful  effort,  the  posterior  distribution  should  yield  a  higher  reliability,  with  reduced 
uncertainty.  This  will  be  shown  later  in  the  example  appendix. 

With  this  technical  foundation  established,  Appendix  B  will  explain  the  PREDICT 
process  steps  required  to  calculate  initial  reliability  (Ro). 


Notes 

1  DoD  3235. 1-H,  “Department  of  Defense  Test  and  Evaluation  of  System  Reliability 
Availability  and  Maintainability-A  Primer,”  March,  1982,  2-2. 

2  DoD  3235. 1-H,  2-2. 

3 

Booker,  Jane  M.,  Los  Alamos  National  Laboratory,  interviewed  by  author,  20 
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4  Ibid. 

5  Ibid. 

6  DoD  3235. 1-H,  2-7. 

7  Booker  interview. 

o 
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line.  Available  at:  http://www.visionengineer.com/mech/monte_carlo_simulation.shtml. 

9  Press,  William  H.,  Saul  A.  Teukolsky,  William  T.  Vetterling,  Brian  P.  Flannery. 
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Philadelphia,  Pennsylvania:  American  Statistical  Association  and  the  Society  for 
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Appendix  B 


PREDICT  -  DEFINING  THE  PROCESS 

PREDICT  has  been  used  successfully  in  automotive  development  and  nuclear 
weapon  physics  packages.  “The  PREDICT  application  is  a  formal,  multidisciplinary 
process  for  estimating  the  perfonnance  of  a  product  when  test  data  are  sparse  or 
nonexistent.”1 


Information  Combination 

PREDICT  combines  diverse  information  sources  to  determine  performance  or 
reliability  of  a  product,  and  tracks  that  perfonnance  or  reliability  as  the  product  changes 
or  as  additional  data  (test  data,  for  example)  becomes  available.  In  the  automotive 
industry,  PREDICT  has  been  used  as  a  development  guide  before,  during,  and  after 
prototyping  and  then  documenting  life  cycle  performance."  This  was  discussed  in 
Chapter  4,  Commercial  Application  of  PREDICT. 

The  key  contribution  PREDICT  brings  to  the  table  is  the  ability  to  “estimate 

reliability  early  in  product  development,  before  costly  design  and  production  decisions 

are  made.”  PREDICT  pulls  together  all  available  information  including  expert 

knowledge,  historical  data,  experience  with  similar  products,  and  computer  simulation. 

With  PREDICT,  these  data  sources  are  gathered  and  combined  to  produce  reliability 

estimates  early  and  through-out  the  product  life  cycle.  PREDICT  can  provide  accurate 
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reliability  estimates  at  the  engineering  concept  phase.  As  the  product  changes  and  /  or  as 
additional  data  become  available,  the  PREDICT  model  can  be  updated  to  provide  a  life 
cycle  track  record  of  the  product  performance  4 

PREDICT  -  What  Is  It? 

A  good  first  step  is  to  state  what  PREDICT  is  not.  PREDICT  is  not  a  management 
approach  like  Total  Quality  Management  or  an  acquisition  strategy  like  Spiral 
Development.  Further,  the  implementation  of  PREDICT  on  a  given  development  effort 
has  not  been  boiled  down  to  filling  in  the  blanks  in  a  checklist  or  software  tool.  On  the 
contrary,  PREDICT  is  a  process,  not  a  software  tool  or  the  latest  acquisition  reform 
buzzword.  PREDICT  captures  expertise  (knowledge)  to  determine  product  or  concept 
performance  early  in  the  change  or  development  process. 

The  PREDICT  process  is  graphically  displayed  in  figure  six.  The  following 
discussion  elaborates  on  the  first  four  steps  of  the  PREDICT  process.  These  four  steps 
yield  the  initial  reliability  estimate,  (Ro).  The  remaining  steps  provide  for  reliability 
updates  as  new  data  become  available  through  the  product  life  cycle. 
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PRE  DIC  T,  Enhuu  «1  Rdiibiljty  MeQudolon 


Figure  6  PREDICT  Process5 
Define  Performance 

Reliability  is  classically  defined  as  the  probability  a  system  does  not  fail  or  functions 
properly.6  In  common  applications,  the  proper  function  of  a  part,  subsystem,  or  system  is 
defined  in  a  specification.  The  PREDICT  implementation  must  clearly  define  these 
specifications  in  language  the  particular  industry  understands.  The  specifications  can 
vary  greatly  from  material  properties  to  the  man-machine  interface,  for  example.  There 
will  likely  be  multiple  performance  definitions  that  define  the  system  performance.  How 
does  a  team  of  statisticians  and  analysts  accurately  define  the  performance  of  complex 
systems  and  subsystems  which  are  very  likely  unfamiliar  to  them? 

A  critical  element  in  building  the  PREDICT  team  is  to  assign  a  small  number  of 
SMEs  to  the  PREDICT  team  as  “insiders,”  (i.e.,  translators  or  advisors).  Insiders  provide 
detailed  knowledge  of  the  system  or  subsystem  under  review  and  insight  on  the  jargon 
used  by  SMEs.  Insiders  are  key  experts  who  “provide  entry  into  their  culture  of  both 
other  experts  and  management,  explain  its  workings  to  analysts,  provide  guidance  on 
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elicitation,  and  motivate  wider  participation  by  other  experts.”7  Insiders  have  personal 
relationships  drawn  from  years  of  experience  working  with  the  engineers  and  scientists 
whose  knowledge  is  the  linchpin  of  the  PREDICT  process.  The  insiders  “bridge  the  gap” 
between  the  analysts  and  the  SMEs,  so  the  system  performance  can  be  defined  in  the 
language  (or  very  likely  the  jargon)  of  the  customer. 

The  PREDICT  process  systematically  defines  performance  specifications  in  terms 
the  engineers  and  designers  understand.  The  SMEs’  knowledge  and  beliefs  about  the 
concept  or  system  dating  back  to  its  origin,  if  needed,  are  documented  for  later  analysis. 

Structure  System 

During  this  step,  a  graphical  representation  of  the  system  and  the  system 
performance  is  developed.  This  is  done  at  the  part,  subsystem,  and  system  level. 
Manufacturing  processes,  environmental  conditions  and  failure  mechanisms  should  also 
be  included.  Representations  such  as  fault  trees,  fish  bone  diagrams  and  reliability  block 
diagrams  are  constructed  in  a  manner  easily  understood. 

A  key  component  of  the  structure  is  defining  the  interrelationships  between  the  parts 
and  processes.  These  interrelationships  must  be  defined  mathematically,  based  on 
knowledge  gained  through  formal  elicitation.  To  capture  complex  interrelationships 
between  parts  and  subsystems,  a  logic-based  model  is  used.  The  goal  of  a  logic-based 
model  is  to  provide  a  “diagrammatic  representation  of  a  complex  set  of  processes  and  /  or 

g 

parts  which  depicts  a  logic  flow  between  these  constituents.” 

A  number  of  methods  are  available  to  build  a  logic-based  model  -  one  or  a 
combination  of  the  examples  listed  below  may  be  used.9  The  key  consideration  should 
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be  the  ability  of  the  model  to  define  the  complex  interrelationship  between  the  parts,  sub¬ 
systems,  and  processes  involved  in  the  problem. 

•  Process  trees  -  shows  how  one  process  flows  into  another 

•  Bayesian  networks  -  incorporates  prior  knowledge  to  detennine  reliability 

•  Weibull  model  -  graphical  representation  of  a  typical  life  cycle  using  a  bathtub 
curve 

•  Probability  networks  -  depicts  the  probabilities  of  various  parts  and  processes 
interaction 

•  Directed  graphic  techniques  -  fault  trees  or  decision  trees  which  depict  the  flow 
of  time  or  force,  for  example,  through  a  problem 

The  bottom  line  is  this  step  “includes  the  formulation  of  mathematical  models  and 

functional  relationships  that  bind  the  parts,  nodes,  and  levels  of  the  system  structure 

together.”10  Without  a  solid  graphical  depiction  of  the  system,  the  PREDICT  analyst  is 

unable  to  determine  where  to  begin  and  where  to  end.  The  system  structure  is  the 

roadmap  for  the  elicitation  step. 


Gather  and  Elicit  Information 

The  work  accomplished  defining  the  system  structure  lays  the  foundation  for  this,  the 
formal  elicitation  step.  Once  the  structure  is  understood,  expert  knowledge  is  gathered  to 
determine  reliability  and  corresponding  uncertainty  for  parts  or  subsystems  prior  to 
design  or  in  the  absence  of  test  data.  For  each  part  or  subsystem,  PREDICT  analysts  use 
a  systematic  approach  to  determine  the  likelihood  of  all  potential  failure  modes.  The  goal 
of  PREDICT  is  to  gather  all  we  know,  and  determine  how  well  we  know  it  (the 
uncertainty).  In  sparse  data  situations,  most  of  this  knowledge  comes  from  the  SMEs. 
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Documentation 


Although  a  “Finalize  Documentation”  step  is  listed  last  in  the  PREDICT  process 
(figure  6),  documentation  is  critical  during  all  steps  of  the  process.  It  is  discussed  here, 
after  the  elicitation  step,  because  in  order  to  accomplish  the  fourth  step  (Calculate  Initial 
Reliability),  data  must  be  documented  in  a  manner  that  allows  the  statisticians  to  do  their 
work. 

Whatever  software  or  database  package  is  used  for  documentation,  it  must  provide 
traceability  for  decisions  giving  the  Who,  What,  When,  and  Why  a  given  decision  was 
made.  To  date,  no  standardized  software  tool  supports  PREDICT.  The  potential 
applications  are  too  diverse  for  a  single  tool  to  facilitate  the  process.  Software  tools  are 
chosen  from  within  the  customer’s  industry  or  business  because  of  the  familiarity  a  given 
team  of  engineers,  for  example,  already  has  with  their  own  tools.  Whether  defining  the 
system  structure  or  conducting  elicitations,  the  PREDICT  team  should  use  documentation 
tools  the  customer  is  already  familiar  with,  to  the  extent  possible. 

Calculate  Initial  Reliability 

This  step  pulls  together  the  results  of  the  structure  and  elicitation  steps.  The  models 
used  for  the  structure  are  populated  with  reliability  estimates  (with  uncertainty  limits) 
from  statistical  analysis  of  elicitations,  test  data,  historical  data,  and  experience.  For  new 
systems,  the  data  will  come  mostly  from  elicitations,  while  a  legacy  system  undergoing 
change  will  use  elicitations  as  well  as  applicable  test  and  historical  data  and  experience. 

The  output  of  this  step  is  an  uncertainty  distribution  for  initial  reliability.  This 
becomes  the  baseline  on  which  to  build  future  updates  as  additional  data  from  test,  for 
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example,  becomes  available.11  As  shown  in  figure  seven,  the  initial  reliability  will  likely 
be  projected  below  the  requirement  with  large  uncertainty. 

Updating  the  Initial  Estimate 

The  remaining  steps  in  the  PREDICT  process  (figure  six)  are  accomplished  as 
additional  data  /  information  become  available.  Results  from  developmental  test  and 
prototyping  should  generate  new  information  which  will  most  likely  increase  reliability 
while  decreasing  uncertainty,  as  shown  in  figure  seven.  A  successful  effort  will  enter 
production  at  or  above  the  required  reliability,  within  the  uncertainty  specified  by  the 
customer.  The  same  progress  can  be  tracked  on  a  legacy  system  undergoing  change  (for 
example,  a  fielded  missile  system  being  upgraded  with  a  new  guidance  package).  Once 
the  upgraded  subsystem  is  fielded,  data  from  operational  testing,  sustainment,  and  aging 
and  surveillance  activities  can  be  incorporated  into  a  PREDICT  reliability  calculation 
using  Bayes’  Theorem.  Appendix  C  provides  a  detailed  mock  example  of  a  PREDICT 
application. 


Tracking  Reliability  Through  System  Lifetime 


Figure  7  Reliability  Through  Life  Cycle12 
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Appendix  C 


PREDICT  Example 

The  following  example1  demonstrates  how  to  calculate  an  initial  reliability  (Ro),  the 
fourth  step  in  the  PREDICT  process  (figure  six).  Also  shown  is  the  result  of 
incorporating  test  data  that  was  not  available  at  the  time  of  Ro,  generating  an  update,  Ri. 
The  purpose  is  to  illustrate  how  expert  knowledge,  combined  with  results  from  successful 
testing,  yields  a  higher  reliability  estimate  with  less  uncertainty.  Note,  this  is  a  mock 
example,  for  illustrative  purposes  only. 

Valve  Operation 

In  this  example,  a  new  design  is  proposed  for  a  valve  assembly,  whose  function  is  to 
pennit  the  flow  of  gas  from  a  gas  reservoir  into  another  pressure  vessel.  Figure  eight 
shows  a  diagram  of  the  valve. 
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Figure  8  Gas  Valve  Cutaway 

The  valve  functions  by  firing  an  actuator,  an  electro-explosive  device  which 
generates  a  high-pressure  gas  that  pushes  a  piston  down  the  inside  of  a  cylinder.  As  the 
piston  moves  through  the  cylinder,  it  cuts  two  tube  caps  opening  a  channel  for  gas  to  flow 
from  a  reservoir  to  its  desired  destination.  The  rear  of  the  piston  is  machined  with  a 
flared  skirt  at  a  specific  angle.  Upon  assembly,  this  flared  skirt  is  slightly  deformed  when 
it  is  pressed  into  the  piston  cylinder,  thus  providing  a  gas  seal.  If  the  piston  functions 
properly,  the  skirt  will  maintain  this  gas  seal  through  the  entire  range  of  piston  motion. 
However,  if  the  skirt  cracks  or  becomes  separated  from  the  piston,  an  undesired  gas  path 
is  created  resulting  in  system  failure.  The  valve  assembly  system  structure  has  nine 
components  of  which  all  are  in  series,  meaning  all  must  function  for  the  valve  to  work,  as 
shown  in  figure  nine.  The  reliability  formula  for  the  valve  assembly  is  also  provided  in 
figure  nine. 


42 


9 

Reliability^,  =  RfR,.  R9  =  rff^ 

i=l 

Figure  9  Valve  Assembly  System  Structure 

R0  -  Initial  Reliability 

In  this  example,  initial  reliability,  (Ro)  is  calculated  using  existing  data  and  expert 
knowledge.  What  is  learned  from  R0  will  be  used  to  guide  the  test  program  by  focusing 
attention  on  areas  where  reliability  is  estimated  as  low  and  /  or  uncertainty  is  high. 

In  the  valve  assembly  system  structure  (figure  nine),  the  component  of  interest  is  the 
component  labeled  “piston  travels  without  skirt  cracking.”  The  probability  of  the  piston 
not  cracking  (Ro  piston)  and  the  corresponding  uncertainty  will  be  calculated.  The  process 
for  calculating  the  first  reliability  update  (Ri  pist0n)  and  its  corresponding  uncertainty  will 
also  be  described. 

The  new  piston  design  has  several  new  features,  but,  to  simplify  the  example,  only 
the  piston  skirt  angle  will  be  considered.  One  reason  for  the  new  piston  design  is 
previous  piston  skirts  were  found  cracked  in  fired  (actuated)  valve  assemblies.  As 
discussed  above,  a  cracked  piston  skirt  leads  to  system  failure  due  to  improper  flow  path. 
The  cracks  were  found  in  pistons  with  certain  skirt  angles.  Smaller  skirt  angled  pistons  in 
different  valve  assemblies  showed  no  cracking,  but  these  smaller  angles  are  less  desirable 
from  a  manufacturing  standpoint. 
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Expert  Elicitation 


Knowledge  about  cracking  and  skirt  angle  behavior  is  not  well  defined,  so  obtaining 
reliability  estimates  for  the  new  piston  skirt  angle  had  high  uncertainty  and  relied  upon 
expert  knowledge  from  valve  designers,  production  personnel,  and  engineering  analysts 
who  ran  finite  element  mechanics  codes  on  the  piston.  Experts  in  these  fields  were  able 
to  provide  their  estimates  of  piston  performance  for  different  skirt  angles  for  this  new 
system  based  upon  their  knowledge,  previous  data,  and  experience.  These  estimates  were 
obtained  in  several  elicitation  meetings,  which  were  one-on-one  and  face-to-face 
interviews  with  each  expert.  Separate  interviews  were  held  so  one  expert  would  not  bias 
another. 

Not  all  the  experts  agreed  on  the  failure  probabilities  for  given  skirt  angles.  As 
shown  in  figure  ten,  the  experts  agreed  up  to  a  specific  skirt  angle  there  was  a  probability 
of  1.0  the  piston  skirt  would  not  crack.  Above  this  skirt  angle  the  experts  differed  on  the 
probability  the  piston  skirt  would  not  crack.  This  uncertainty  region  is  shown  in  red  in 
figure  ten.  The  uncertainty  region  disappears  at  a  higher  skirt  angle,  where  the  experts 
once  again  agreed  on  the  probability  of  the  piston  skirt  not  cracking.  The  proposed  piston 
skirt  angle,  indicated  by  the  blue  dashed  line,  falls  within  this  uncertainty  region  as 
illustrated  in  figure  ten. 
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Figure  10  Skirt  Angle  Probabilities 


Rii  Piston  -  Initial  Reliability 

From  figure  ten,  the  reliability  interval  for  the  desired  skirt  angle  is  roughly  from  0.8 
to  1.0.  This  is  the  lower  and  upper  bound  of  the  intersection  of  the  blue  dashed  line  with 
the  red  uncertainty  region.  This  interval  represents  the  probability  of  not  cracking  for  a 
given  condition  (skirt  angle),  drawn  from  the  knowledge  of  the  experts.  The  experts 
agreed  to  use  the  Beta  probability  density  function  (PDF)  in  figure  eleven  to  represent  the 
likelihood  the  piston  skirt  would  not  crack  in  the  interval  from  0.8  to  1.0.  The  vertical 
axis  of  the  PDF  is  not  a  probability  but  a  density,  which  represents  a  relative  likelihood  of 
occurrence  of  each  value  along  the  horizontal  axis.  The  figure  eleven  PDF  curve  shows 
the  desired  skirt  angle  is  most  likely  to  have  a  .97  reliability  of  not  cracking. 
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Figure  11  Beta  PDF  for  Desired  Skirt  Angle 

This  same  fitting  of  a  PDF  curve  must  be  done  for  each  of  the  potential  skirt  angles 
in  the  uncertainty  region  from  figure  ten.  These  Beta  PDFs  are  combined  or  summed  to 
generate  the  Beta  PDF  curve  shown  in  figure  twelve.  The  combination  of  distributions  is 
done  with  a  Monte  Carlo  simulation. 
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Reliability 

Figure  12  Beta  PDF  Summed  Over  All  Skirt  Angle  PDFs 

The  mean  for  figure  twelve  is  0.73,  with  a  standard  deviation  of  0.18.  Doubling  the 
standard  deviation,  this  corresponds  to  a  reliability  range  of  [.36,  1.09].  Because  the 
upper  range  is  greater  than  1.0,  a  90  percent  probability  interval  is  used  instead.  This 
means  90  percent  of  the  time,  the  reliability  of  the  piston  will  be  within  the  stated  range. 
Ro  Piston  equals  0.73,  with  a  90  percent  probability  interval  for  reliability  of  [0.37,  0.97]. 
The  analysis  of  the  knowledge  from  the  SMEs  yielded  a  low  reliability  with  large 
uncertainty. 
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The  same  process  would  be  followed  to  calculate  the  reliability  and  uncertainty  of  all 
nine  components  in  the  valve  assembly  system  structure  (figure  nine).  To  determine  the 
reliability  of  the  valve  assembly,  a  Monte  Carlo  simulation  would  be  run  to  combine  the 
reliabilities  and  uncertainties  from  each  of  the  components  of  the  valve  assembly. 

Ri  Piston  -  First  Reliability  Update 

A  test  program  was  needed  for  the  piston  to  increase  the  reliability  estimate  and 
decrease  its  associated  uncertainty.  PREDICT  determined  the  desired  piston  skirt  angle 
was  within  a  region  of  uncertainty  and  focused  the  test  program  on  the  red  zone  in  figure 
ten.  Once  data  from  the  piston  skirt  angle  test  become  available,  Ro  piston  can  be  updated 
using  Bayes’  Theorem. 

With  Bayes’  Theorem,  the  prior  distribution  (R0  piston,  green  in  figure  twelve)  is 
combined  with  the  test  data  results.  The  test  program  resulted  in  five  successes  out  of 
five  trials.  This  yields  a  90  percent  confidence  interval  of  [.63,  1.0],  with  a  median  value 
of  0.87.  Median  is  used  versus  mean  because  the  test  data  reflected  a  highly  skewed  (not 
symmetric)  binomial  distribution.  Bayes’  Theorem  is  now  used  to  combine  the  prior 
distribution  from  the  experts,  with  test  data  to  generate  the  posterior  distribution,  Ri  Piston 
(blue  in  figure  thirteen). 
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Figure  13  Beta  PDF  for  Desired  Skirt  Angle 

For  Ri  Piston,  the  mean  is  0.87  with  a  90  percent  probability  interval  of  [0.67,  0.99]. 
Combining  the  Ro  piston  with  the  test  data  increased  the  calculated  reliability  from  0.73  to 
0.87  and  the  standard  deviation  decreased  from  0.18  to  0.10.  Likewise,  the  90  percent 
probability  interval,  [0.37,  0.97],  was  improved  to  [0.67,  0.99].  This  represents  a 
significant  improvement  in  reliability  and  reduction  in  uncertainty.  Note  that  if  some  or 
all  of  the  tests  were  failures,  Ri  Piston  would  reflect  less  improvement  or  a  reduction  in 
reliability  with  corresponding  impacts  to  uncertainty. 

This  example  demonstrates  the  PREDICT  contribution  to  assessing  reliability.  The 
early  calculation  of  reliability  documented  the  expert  knowledge  and  focused  the  test 
program  on  an  area  of  uncertainty  in  the  gas  valve  design.  The  test  program  produced 
results  consistent  with  the  SME’s  knowledge  leading  to  significantly  improved  reliability 
and  reduced  uncertainty. 

Notes 

1  Booker,  Jane  M.,  Los  Alamos  National  Laboratory,  interviewed  by  author,  20 
January,  2005. 


48 


Appendix  D 


Classified  Report  Access 


For  access  to  the  B61-7,  -11  Life  Extension  Program  Reliability  classified  report, 
contact  the  LANL  POC  listed  below  to  determine  required  clearance  level  and 
need-to-know. 

Name:  Mr.  Gerald  Schotik 
Office  Symbol:  ESA-WSE 
Phone  Number:  505  664-0273 
Official  Mailing  Address: 

University  of  California 
Los  Alamos  National  Laboratory 
PO  Box  1663,  MS  P234 
Los  Alamos  NM,  87545 
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